Harvesting machine capable of automatic adjustment

ABSTRACT

A harvesting machine capable of automatic adjustment, comprising a plurality of acoustic material flow sensors, a control system, a processor, and application software, wherein the plurality of acoustic material flow sensors are mounted internal to the harvesting machine at points of crop material flow and are capable of sensing an amount of crop material passing by them. The control system operates in a cause-and-affect mode for interactively enabling manual or automatic responses to Mass Material Distribution (MMD) information and equipment-related performance parameters. An interactive combine control method is also provided.

CROSS REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation-in-part of and claims priorityin U.S. patent application Ser. No. 14/853,949, filed Sep. 14, 2015, nowU.S. Pat. No. 9,629,308, issued Apr. 25, 2017, which claims the benefitof U.S. Provisional Patent Application No. 62/049,616, filed Sep. 12,2014, and is a continuation-in-part of and claims priority in U.S.patent application Ser. No. 13/843,029, filed Mar. 15, 2013, now U.S.Pat. No. 9,330,062, issued May 3, 2016, which is a continuation-in-partof and claims priority in U.S. patent application Ser. No. 13/046,549,filed Mar. 11, 2011, now U.S. Pat. No. 8,950,260, issued Feb. 10, 2015,and is also a continuation-in-part of and claims priority in U.S. patentapplication Ser. No. 13/678,441, filed Nov. 15, 2012, now U.S. Pat. No.9,474,208, issued Oct. 25, 2016, which claims the benefit of U.S.Provisional Patent Application No. 61/560,201, filed Nov. 15, 2011, allof which are incorporated herein by reference in their entireties.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates generally to the field of precision agriculture,and specifically to a method and system of automating the set up andadjustment of a combine or similar agricultural vehicle.

2. Description of the Related Art

There is a desire to automate the adjustment of a combine (also known asa “harvester”) so that very little human know-how is required to operatethe vehicle. This would enable the hiring of unskilled labor to operatethe combine, reducing the cost to the farmer. It could also increase theefficiency of the harvesting process, and therefore the crop yield andmachine productivity.

Attempts have been made to automate combines already, but the sensorsthat have been used to sense the necessary conditions, such as load onthe machine and the cleanliness of the crop being harvested, areinadequate for the job.

What is needed in the art is a method and system for automating acombine that relies on advanced sensors that can detect and monitor theamount and quality of material moving through the combine at any time.

SUMMARY OF THE INVENTION

This invention describes a method and system for the automaticadjustment of a combine, or for providing directives to an operator todo the same.

In one aspect of the invention, a harvesting machine capable ofautomatic adjustment is provided, comprising a plurality of materialflow sensors, a control system, a processor, and software, wherein thematerial flow sensors are capable of sensing an amount of crop materialpassing by them, wherein the control system is capable of adjusting aset of internal elements of the harvesting machine, wherein the softwareis hosted on the processor, wherein the processor is operatively coupledto the control system and the plurality of material flow sensors,wherein the software uses information sensed by the plurality ofmaterial flow sensors to determine if the set of internal elements ofthe harvesting machine are set for optimal machine performance, andwherein the software sends commands to the set of internal elements ofthe harvesting machine in order to improve the machine performance.

In another aspect of the invention, a material flow sensor is provided,comprising an acoustic chamber, an impact plate and a housing, apneumatic impulse line, a microphone, and an electronics module, whereinthe acoustic chamber and the microphone are connected by the pneumaticimpulse line, wherein the housing is shaped so as to direct sound wavescreated by at least one object striking the impact plate into thepneumatic impulse line, wherein the sound waves move through thepneumatic impulse line into the microphone, wherein the microphonedetects the sound waves and converts them into an electrical signal,wherein the microphone is electrically connected to the electronicsmodule, and wherein the electronics module analyzes the electricalsignal and converts it into a representative mass of the at least oneobject striking the impact plate.

In yet another aspect of the invention, a grain quality sensor isprovided, comprising a lens, a filter, a photosite array, at least oneillumination source, and an electronics module, wherein the filter isplaced between the lens and the photosite array, wherein theillumination source directs light containing a known set of wavelengthsonto a crop sample, wherein the lens picks up any light reflected by thecrop sample and directs it into the filter, wherein the filter allowslight to pass into different parts of the photosite array such thatcertain locations on the photosite array only get certain frequencies ofthe reflected light and other certain locations on the photosite arrayonly get other certain frequencies of the reflected light, wherein theelectronics module is electrically connected to the photosite array andcapable of determining which parts of the photosite array received lightand what frequency the light received was, wherein the electronicsmodule can analyze the optical data received by the photosite array,wherein the analysis of the optical data is used to determine thecomposition of different parts of the crop sample, and wherein no imageof the crop sample is ever created.

In yet another aspect of the invention, a method of creating imageswhich contain only a portion of the photographed subject matter isprovided, the method comprising the steps of placing a color filter on aphotosite array, focusing light on the color filter, capturing photonsin a photosite array, analyzing and processing the information gatheredon the photons captured, determining the color information representedby individual photosites in the photosite array, altering the colorinformation so as to delete information from photosites representingcolors of a certain frequency, and creating an image from the remainingcolor information, wherein an image can be created that contains onlysome of the original elements present in the photographed subjectmatter.

In yet another aspect of the invention, a crop quality sensor isprovided, comprising an illumination source, an imaging device, aprocessor; and software executing on the processor, wherein theillumination source is shone onto a crop sample, wherein the crop sampleis such that individual kernels of the crop have a shiny outer casingand a dull inner surface when broken open, wherein an image is takenwith the imaging device of the illuminated crop sample, wherein thesoftware is executing on the processor, wherein the software is used toanalyze the image to identify the outlines of individual kernels and toidentify which of those outlines contain a specular highlight, andwherein the presence of a specular highlight within an outline isindicative that that kernel is whole and unbroken and the absence ofsuch a specular highlight is indicative of a broken kernel.

In yet another aspect of the invention, a yield sensor is provided,comprising an acoustic chamber comprising an impact plate and a housing,a pneumatic impulse line, a microphone, and an electronics module,wherein the acoustic chamber and the microphone are connected by thepneumatic impulse line, wherein the housing is shaped so as to directsound waves created by at least one object striking the impact plateinto the pneumatic impulse line, wherein the sound waves move throughthe pneumatic impulse line into the microphone, wherein the microphonedetects the sound waves and converts them into an electrical signal,wherein the microphone is electrically connected to the electronicsmodule, and wherein the electronics module analyzes the electricalsignal and converts it into a representative mass of the at least oneobject striking the impact plate.

In yet another aspect of the invention, a crop mass predictive sensor isprovided, comprising an imaging device, a LIDAR, a first radar emittinga frequency of energy that is absorbed by plant mass, and a second radaremitting a frequency of energy that passes through plant mass withoutbeing absorbed, wherein the imaging device, LIDAR, first radar, andsecond radar are focused on the crop material in front of anagricultural vehicle, and the information gathered from each of thesecomponents is used to calculate an estimated mass for the crop materialthat is about to enter the agricultural vehicle.

In yet another aspect of the invention, a crop mass predictive sensor isprovided, comprising an imaging device, a LIDAR, a first radar emittinga frequency of energy that is absorbed by plant mass, a second radaremitting a frequency of energy that passes through plant mass withoutbeing absorbed, and a location sensor, wherein the imaging device,LIDAR, first radar, and second radar are focused on the crop material tothe side of an agricultural vehicle, and the information gathered fromeach of these components is used to calculate an estimated mass for thecrop material, and the estimated mass is stored along with a currentlocation from the location sensor for subsequent use, by the currentmachine, or transmitted to a separate machine for its use.

In yet another aspect of the invention, a method of determining theshape of at least a portion of a surface relative to a designatedexternal point of reference is provided, comprising the steps of placingan imaging device at the designated external point of reference suchthat it can take an image of the at least a portion of a surface,projecting a straight line onto the at least a portion of a surface froma point that is offset by a predetermined angle from the designatedexternal point of reference, taking an image of the at least a portionof a surface with the imaging device, and analyzing the image todetermine the shape of the at least a portion of a surface, wherein theanalysis comprises determining the apparent distance from the imagingdevice to a series of points along the projected line based on theperceived shape of the line when viewed from the designated externalpoint of reference.

In yet another aspect of the invention, a mobile device for use as auser interface for an agricultural vehicle is provided, wherein themobile device can receive messages from and transmit messages to thecontrol system of the agricultural machine.

In yet another aspect of the invention, a harvesting machine capable ofproviding recommendations to an operator comprising a plurality ofmaterial flow sensors; a control system, a display, a processor, andsoftware, wherein the material flow sensors are capable of sensing anamount of crop material passing by them, wherein the control system iscapable of adjusting a set of internal elements of the harvestingmachine, wherein the software is hosted on the processor, wherein theprocessor is operatively coupled to the control system and the pluralityof material flow sensors, wherein the software uses information sensedby the plurality of material flow sensors to determine if the set ofinternal elements of the harvesting machine are set for optimal machineperformance, and wherein the software sends recommended control settingsto the display, whereby the operator uses the recommended controlsettings as necessary to change the settings on the harvesting machine'sinternal elements for optimal performance.

In yet another aspect of the invention, a method of estimating theamount of crop mass entering a harvesting machine is provided,comprising the steps of attaching potentiometers to the front feedroller of a the harvesting machine and using the potentiometers tomeasure the magnitude of deflection of the front feed roller as cropmass is pushed under the front feed roller, causing it to rise.

In yet another aspect of the invention, a method of estimating the massof crop entering into a grain tank from a clean grain elevator on aharvesting machine is provided, comprising the steps of mounting atleast one load sensor on an upper bearings of a conveyor belt movinggrain through the clean grain elevator into the grain tank, using theload sensors to measure the load on the conveyor belt when no grain ispresent in the clean grain elevator, using the load sensors to measurethe load on the conveyor belt when grain is moving through the cleangrain elevator, and comparing the load with no grain present to the loadwhen grain is present to determine the mass of crop moving through theclean grain elevator.

The features, functions, and advantages can be achieved independently invarious embodiments of the present invention or may be combined in yetother embodiments in which further details can be seen with reference tothe following description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings constitute a part of this specification and includeexemplary embodiments of the invention illustrating various objects andfeatures thereof, wherein like references are generally numbered alikein the several views.

FIG. 1 is a block diagram of a combine showing the various components ofthe combine involved in the present invention, along with the placementof sensors needed for the present invention.

FIG. 2A shows a block diagram of a camera or “imaging device” from theprior art and how it is used to capture an image.

FIG. 2B shows a block diagram of an optical data capture sensor from thepresent invention and how it is used to determine the composition of acrop sample without taking images.

FIG. 2C is a perspective view of one embodiment of an optical datacapture sensor form the present invention.

FIG. 2D is a side view of one embodiment of an optical data capturesensor form the present invention.

FIG. 2E illustrates how the present invention can create “partialimages” or “layer images” that eliminate visual elements present in theoriginal subject matter.

FIG. 2F illustrates an alternate embodiment of a grain quality sensorthat detects damaged grain by detecting the lack of specular highlightson certain kernels.

FIG. 2G shows an image that has been processed to highlight “bright”spots or specular highlights.

FIG. 3A is a flowchart showing the processes used to create the numericvalues and other outputs of the optical data capture sensor of thepresent invention.

FIG. 3B illustrates one embodiment of an algorithm for analyzing valuesin a photosite array to determine the content of a grain or crop sample.

FIG. 3C describes how the demosaicing process of the prior art works.

FIG. 3D illustrates how introducing the demosaicing process of the priorart into the process of FIG. 3A may improve performance.

FIG. 4A shows the clean grain elevator of a typical combine and thesensors associated with the clean grain elevator as defined for use inthe present invention.

FIG. 4B shows an alternate mounting location and system for the opticaldata capture sensor (grain quality sensor) of the present invention.

FIG. 5 shows the main functional components of one embodiment of alook-ahead sensor of the present invention.

FIG. 6A shows a top view of a combine showing how the radar-basedcomponents of the look-ahead sensor of FIG. 5 would work to predictincoming crop load.

FIG. 6B shows a top view of a combine showing how the LIDAR-basedcomponent of the look-ahead sensor of FIG. 5 would work to predictincoming crop load.

FIG. 6C shows a top view of a combine using an alternate embodiment ofthe look-ahead sensor of the present invention which looks to the sideof the combine, instead of ahead of the combine.

FIGS. 6D through 6J illustrate an alternate embodiment of the LIDARportion of the crop mass sensor 506.

FIG. 7A shows one embodiment of an application user interface page forthe present invention as displayed on a mobile computing device.

FIG. 7B shows another embodiment of an application user interface pagefor the present invention as displayed on a mobile computing device.

FIG. 7C shows yet another embodiment of an application user interfacepage for the present invention as displayed on a mobile computingdevice.

FIG. 8 shows a series of combine adjustments that may be made by thepresent invention, as well as the system inputs that are used todetermine what adjustments are to be made.

FIG. 9 shows one embodiment of a control system architecture for thepresent invention.

FIGS. 10A through 10N are a series of flowcharts which capture logicthat may be used by the present invention to determine which combineadjustments to make.

FIG. 11 shows an alternative embodiment control system including atriangular graphic user interface (GUI) and material mass distribution(MMD) graphs. Data are displayed in imperial units.

FIG. 11A shows the GUI of FIG. 11 with an enlarged touchscreen portionfor operator control of predetermined operating parameters.

FIG. 12 shows the GUI with different data displayed. The data aredisplayed using metric units.

FIG. 13 shows the GUI in an alternative operating mode with MMD factorsaltered from FIG. 11 whereby optimum crop (e.g., grain) quality andminimum loss are emphasized with the processor focusing on thesevariables.

FIG. 13A shows the GUI of FIG. 13 with an enlarged, triangulartouchscreen portion for operator control of predetermined operatingvariables for optimizing certain parameters.

FIG. 13B shows the GUI of FIG. 13 with a pop-up window showing an imageof grain material in the combine (harvester).

FIG. 14 shows an alternative embodiment of the system with a modifiedGUI subsystem including an increase-decrease switching arrangement foradjusting predetermined operating parameters and other GUI features andfunctions.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

I. Introduction and Environment

As required, detailed aspects of the present invention are disclosedherein, however, it is to be understood that the disclosed aspects aremerely exemplary of the invention, which may be embodied in variousforms. Therefore, specific structural and functional details disclosedherein are not to be interpreted as limiting, but merely as a basis forthe claims and as a representative basis for teaching one skilled in theart how to variously employ the present invention in virtually anyappropriately detailed structure.

Certain terminology will be used in the following description forconvenience in reference only and will not be limiting. For example, up,down, front, back, right and left refer to the invention as orientatedin the view being referred to. The words, “inwardly” and “outwardly”refer to directions toward and away from, respectively, the geometriccenter of the aspect being described and designated parts thereof.Forwardly and rearwardly are generally in reference to the direction oftravel, if appropriate. Said terminology will include the wordsspecifically mentioned, derivatives thereof and words of similarmeaning.

With reference now to the drawings, and in particular to FIGS. 1 through10N thereof, a new method and system of automating the adjustment of acombine embodying the principles and concepts of the present inventionwill be described.

In general terms, the present invention will automate the adjustment ofa combine by following a series of steps, including:

-   -   A. Equipping an agricultural combine with new sensors placed        throughout the combine to sense the state of the material        flowing through the machine at any given time and location        internal to the machine.    -   B. Collecting and analyzing the data gathered on the material        flow.    -   C. Determine which adjustments could be made to internal        components of the combine (based on the analysis of the data        collected from the sensors) to optimize the combine's        performance.    -   D. Automatically make the adjustments to the combine components        to optimize the combine's performance, OR    -   E. Make recommendations to the operator of the combine, or        provide them with actionable data, so that they may make manual        adjustments to the combine components to optimize the combine's        performance.

FIGS. 1 through 6C describe the types of sensors used to complete StepsA and B of the above process. FIGS. 8 through 10N describe the stepsneeded to complete Steps C through E of the above process. FIGS. 7Athrough 7C describe the invention's optional user interface which can beused to make the recommendations to the operator as discussed in Step Eof the above process.

II. Crop Material Flow Sensor Type and Placement

The key to the present invention is to be able to detect the status ofthe machine (the combine) at any given point, especially to havedetailed information on the flow of crop material through the combinesystem and the condition of the crop material.

At optimal/ideal performance, the crop material collected (harvested) bya combine would be as close to 100% “clean grain” (the “grain” is theseeds of the crop being harvested) as possible with little to no crackedgrain (grain that has been damaged, sometimes by the harvesting processitself) and little to no “material other than grain,” often referred toby those skilled in the art as MOG. Like the phrase “material other thangrain” implies, MOG is any material that is moved through the combineduring harvesting that is not grain. MOG may include things like rocks,dirt, trash, straw and chaff (plant matter that is something other thanthe grain, such as the dry protective casings of seeds and grains, partsof the stems, flowers, leaves, etc.

Improper settings of internal components of a combine harvesting machinecan result in an increase in cracked grain and/or MOG, which lowers thevalue of the harvested crop by adding weight and volume to the harvestedcrop without adding additional value, or by otherwise reducing thequality of the grain. Improper settings can also result in clean grainbeing lost out the back of the combine, reducing yield.

For instance, the crop being harvested is collected by the combine andfed toward a spinning cylinder (called a “rotor”) which spins thematerial against one or more curved metal gratings (called “concaves”).The concaves are shaped to match the curve of the rotor and can be movedfarther from and closer to the rotor as needed. As the rotor carries thecrop material past the concaves, the crop material is threshed as it ismoved over and impacts the concaves, knocking the seeds (the grain)loose from the rest of the plant. The spacing between the rotor andconcave can be adjusted based on the crop type and the size of the grainbeing harvested (and other factors, such as crop load). If the concaveis too close to the rotor, however, or if the rotor speed is too fast,the grain can be damaged and cracked, which makes it more likely to belost in the harvesting process (more likely to be blown away with thechaff in the harvesting process) and also introduces problems inhandling and storage of the grain, including harboring insects andincreasing mold growth, as well as reducing the quality of the grain(for example, reducing protein content). Having the concave too close tothe rotor can also over-thresh the grain, increasing the amount of MOGin the grain that passes through the concaves.

Therefore, if there was a way to detect the percentage of cracked grainthat winds up in the clean grain tank during harvesting, then it wouldbe possible to correct the rotor speed or the rotor-to-concave spacingin real time, during the harvesting process, to minimize the percentageof cracked grain.

This is just one example of a combine adjustment that can be made aspart of the present invention. Other examples will become evidentthroughout the remainder of this specification.

Turning now to FIG. 1, we will discuss the components of a combine 500in additional detail, as well as the various types of sensors that canbe added to the combine 500 in order to implement the present invention.A combine 500, also known as a combine harvester, or simply a harvester,is an agricultural machine that cuts, threshes, and cleans a grain cropin a single machine (in a single operation). It is typicallyself-propelled (a vehicle and not an implement) that is driven into andthrough a crop at harvest time. The operation and working components ofa traditional combine 500 are well known in the prior art and thisspecification will not address all elements of a combine, but willaddress those which are new and/or important to the operation of thepresent invention.

In FIG. 1, a combine 500 has a cab 100 where the operator of the vehicleis housed, and the cab 100 is typically located on what is considered tobe the front of the combine 500 (the direction of forward travel). Atthe very front of a combine 500, a removable header 523 (see FIGS.6A-6C, header not included in FIG. 1) pushes into the crop in thedirection of forward travel and cuts the crop and pulls it into thefeeder housing 111. A typical header 523 has a reciprocating knifecutter bar for cutting the plants near the ground and a revolving reelto cause the cut crop to fall back into the feeder housing 111. Otherversions of combines may use a “pick-up header” instead of a cuttingheader for crops that are cut by a separate machine and placed intowindrows that are later picked up by the combine with such a header. Thetype of header is not pertinent to the present invention, and theexample shown herein should not be considered limiting. The feederhousing 111 contains a conveyor chain 112 or similar mechanism to pullthe cut crop up into the combine for threshing.

One of the important pieces of information for a self-adjusting combineis to know the load seen on the conveyor chain 112, as early as possiblein the harvesting process, as crop moves into the feeder housing 111.Therefore, one or more potentiometers 120 are mounted on the front feedroller to measure the amount of deflection seen at this location. Thematerial pushing into the feeder housing 111 will actually push up onthe conveyor chain 112 mechanism, which “floats” up and down as theamount of material changes. The conveyor chain 112 mechanism typicallycan detect when one side of the feeder housing 111 has more materialthan the other, as both sides of the conveyor chain 112 float separatelyand therefore the separate sides are deflected upward based on theamount of material under each side, and the deflection can be translatedinto amount of mass, or load. In the typical embodiment, there is atleast one potentiometer per side on the conveyor chain 112 mechanism,such that the deflection of each side can be measured independently.

This information can be digitized and sent to other locations on thecombine 500 for use in combine adjustment (as well as other functions).

The crop material is delivered by the conveyor chain 112 to the feedaccelerator 110, which is a rotating drum covered in paddles that pullsthe crop material up into the machine, delivering it into the threshingassembly 116. The threshing assembly 116 includes a rotor 103 and one ormore concaves 103A. The rotor 103 is a spinning cylinder withprojections, such as paddles (also known as threshing elements),arranged in the shape of the inclined plane of an auger, on it such thatis will push the crop material through the combine from the front end ofthe rotor 103 to the back end of the rotor 103. The crop material ispulled through the threshing assembly 116 by the spinning motion of therotor 103, and, as it moves from front to back, the crop material isdragged across the concaves 103A, causing the crop material to bethreshed. The concaves 103A are metal gratings with holes through whichthreshed grain (the seeds that are pulled or shaken off of the cropmaterial) may drop. The material that passes through the concaves 103Adrop into the cleaning shoe 117, where the crop material is furtherprocessed to separate the clean grain from the chaff before it iscollected.

In the embodiment shown in FIG. 1, a series of crop material sensors 104are placed on the bottom side of the concaves 103A. These crop materialsensors 104 can detect the amount of material dropping on them and can,in the preferred embodiment, distinguish between grain and MOG. Thesecrop material sensors 104 may be any type of appropriate sensor fordetecting the impact of particles, including piezoelectric sensors,optical sensors, and mechanical sensors, but in the preferred embodimentare acoustic sensors which can detect the sound of material impactingthe sensors and ideally distinguish between the heavier sounds of grainhitting the sensor and the lighter sounds of chaff hitting the sensors.

It is helpful to know the load on the rotor 103 in order to properlyadjust the combine settings. The “rotor load” is the measure of thepressure put on the rotor 103, and one method of measuring this rotorload is to place a sensor on the rotor pulley actuator 124 which canmeasure the differences in load as the rotor pulley spins the rotor. Therotor load is calculated based on load on the rotor pulley actuator 124and communicated to the combine system to use in determining the combinesettings.

After the crop material passes through the rotor 103 and the concaves103A, it falls down into the cleaning shoe 117. The cleaning shoe 117typically includes a chaffer 108 and a sieve 106. The chaffer 108 andthe sieve 106 are “filters” that typically have adjustable-size openingsin them and which further aid in the separation of grain from MOG. Thechaffer 108 typically has larger openings than the sieve 106, and so thechaffer 108 will allow larger pieces of crop material to pass through tothe sieve 106. As the crop material falls on the chaffer 108 and sieve106, further separation of the material occurs. Forced air generated byone or more fans 113 is propelled through channel 109 and directed upthrough the chaffer 108 and the sieve 106. The air will carry lightermaterial such as chaff up and out of the back of the combine 500 to bedispersed on the ground.

A rotor loss sensor 107 will detect the amount of material that fallsfrom the back of the rotor (meaning it was not completely threshed as ittraveled along the rotor). This rotor loss sensor 107 may be anyappropriate sensor that detects the impact of crop material, and whichcan, in the preferred embodiment, distinguish between grain and MOG. Therotor loss sensor 107 may be any type of appropriate sensor fordetecting the impact of particles, including piezoelectric sensors,optical sensors, and mechanical sensors, but in the preferred embodimentis an acoustic sensor which can detect the sound of material impactingthe sensors at a minimum and, ideally, distinguish between the heaviersounds of grain hitting the sensor and the lighter sounds of chaffhitting the sensors.

At the back end of the chaffer 108 is a grain loss sensor 105. In thepreferred embodiment, the grain loss sensor 105 is a sensor usingacoustic sensor technology, which can detect the sound of materialimpacting the sensor and ideally distinguish between the heavier soundsof grain hitting the sensor and the lighter sounds of chaff hitting thesensors. The purpose of the grain loss sensor 105 is to detect theamount of clean grain that is being lost out of the back of the combine500.

At the back end of the sieve 106 is a tailings sensor 119. In thepreferred embodiment, the tailings sensor 119 is a sensor using acousticsensor technology, which can detect the sound of material impacting thesensor and ideally distinguish between the heavier sounds of grainhitting the sensor and the lighter sounds of chaff hitting the sensors.The purpose of the tailings sensor 119 is to detect the amount oftailings that falls out of the back of the cleaning shoe 117. Inharvesting, “tailings” are a mixture of grain and the mature vegetationon which the grain grows, and, with respect to the combine, the tailingsrepresent the crop material that falls out the back of the cleaning shoe117. In a typical combine, the tailings will be given a “second chance”,where they are collected by a tailings auger 115, which delivers thetailings to a tailing elevator (not shown in drawing) to be transportedback to the rotor 103 for another attempt at threshing.

The heavier grain that is successfully threshed after traveling throughthe rotor 103 and concaves 103A and the cleaning shoe 117 will fall offthe front end of the sieve 106 rather than being blown back by the aircoming from the fan 113. The grain falling off the front end of thesieve 106 will impact a clean grain sensor 118. In the preferredembodiment, the clean grain sensor 118 is a sensor using acoustic sensortechnology, which can detect the sound of material impacting the sensorand ideally distinguish between the heavier sounds of grain hitting thesensor and the lighter sounds of chaff hitting the sensors.

After impacting the clean grain sensor 118, the clean grain will dropinto the clean grain auger 114 and be transported to a clean grainelevator 400 (not shown in this figure but presented in FIG. 4) where itis delivered to the grain tank 101.

Eventually, the grain captured in the grain tank 101 will be offloadedto an agricultural cart or vehicle. This offloading is done through theoffload auger 102.

It should be noted that sensors 104, 105, 107, 118, and 119, areintended to be acoustic material flow sensors in the preferredembodiment, similar to the energy sensing acoustic technology (ESAT)sensors manufactured by Appareo systems, including those disclosed inWO/2012/125575, the latter publication incorporated herein by referencein its entirety, or variants thereof.

An acoustic material flow sensor for a harvesting machine might comprisean acoustic chamber with an impact plate and a housing, a pneumaticimpulse line, a microphone, and an electronics module. The housing ofthe acoustic material flow sensor is shaped so as to direct sound wavescreated by crop matter that is striking the impact plate into apneumatic impulse line connected to the chamber. Once the sound wavesenter the pneumatic impulse line, they travel down the line into amicrophone connected to the other end of the pneumatic impulse line.

The microphone then detects the sound waves and converts them into anelectrical signal that is a representation of a “sound power” derivedfrom the energy of the sound waves collected. The electronics moduleanalyzes the electrical signal and converts it into a representativemass of the crop matter striking the impact plate. This may be done by aspecialized audio processor, designed specifically for the analysis ofaudio signals, such as a processing chip designed for use inmusic-related applications.

The acoustic material flow sensor may also be able to analyze thefrequencies of the sounds generated by crop matter striking the impactplate, and determine if material of largely different densities isstriking the plate. Crop matter that is moving through a harvestingmachine often contains “material other than grain”, or MOG, which may berocks, soil, plant matter other than seed, etc. By distinguishingbetween sound waves representing different densities of crop matter, anapproximate percentage of MOG contained in the crop matter can bedetermined.

However, these material flow sensors may comprise sensors of a varietyof different structures and/or types, as would be known by one skilledin the art.

The purpose of FIG. 1 is to identify the various components of a combineand the variety of sensors needed to detect material flow through thecombine at various points. Some of these sensors already exist in theprior art to collect information for use in other subsystems of acombine, and other sensors are new to the art and these new-to-the-artsensors will be described in additional detail in the remaining figuresof this specification.

It should be noted that FIG. 1 represents one possible embodiment of acombine and is not intended to be limiting. For example, some combinesplace the rotor such that it is perpendicular to the direction oftravel, rather than parallel to it. Some of the sensors described hereinmay be omitted without differing from the intent of the presentapplication.

In addition to the sensors described in the previous section and asshown on FIG. 1, there are additional sensors that sense items otherthan the flow of material through the interior of the combine. Thesesensors, described in the following sections, include a grain qualitysensor, a look-ahead crop mass sensor, a yield sensor, and a moisturesensor.

III. Grain Quality Sensor

The following section, including the discussion of FIGS. 2B through 4,presents a novel grain quality sensor for use in gathering data neededfor the combine automation system and method. FIG. 2A shows a blockdiagram of a camera or “imaging device” from the prior art and how it isused to capture an image.

The concept behind a grain quality sensor is to somehow examine a sampleof crop material from the clean grain tank of a harvester such as thatshown in 101 on FIG. 1 to determine the percentage of (1) damaged grain,(2) material other than grain, and (3) clean grain. Damaged grain isgrain or seeds of a crop for which the outer casing has been damaged,exposing the endosperm (the inside of the seed). Grain can be damaged bythe harvesting process itself if the adjustments on the combine are notoptimized. For instance, if the distance from the rotor to the concavegratings is too close, the grain can be caught between rotor and concaveand threshed too “violently”, causing damage to the outer casing of thegrain/seed. Material other than grain, or MOG, as has been previouslyexplained, is any plant material other than the seed, and can alsoinclude foreign matter such as rocks, soil, and other plant matter (suchas weeds). Clean grain consists of undamaged grain/seed and no MOG.

By determining the percentages of damaged grain, MOG, and clean grain ina sample of harvested material, a control system for a combine can workto make automated adjustments to internal settings such as the distancefrom rotor to concave to improve the percentages of clean grain.

One way to analyze a grain sample to determine these percentages is todo it by image analysis. Several inventions in the prior art use adigital camera to take an image of a sample of grain and then analyzethat image to search for cracked grain and MOG.

FIG. 2A shows a block diagram of a camera or “imaging device” from theprior art and shows how it is used to capture an image for analysis. Acrop sample 200 contains a combination of clean grain 201, damaged grain202, and MOG 203. Prior art inventions use a camera or similar imagingdevice 220 to capture an image 240 of the crop sample 200. The imagingdevice 220 comprises a lens 204, a color filter 260, a photosite array209, and a series of functional blocks which are a mix of electronichardware and firmware. There is a set of analog electronics 205 forpowering and reading the photosite array 209, an analog to digitalconverter 206 for converting the analog voltage values read from theanalog electronics 205 into digital values, a “demosaicing” process 207which is required to compensate for the introduction of the color filter260 (needed to produce an image with accurate color reproduction),digital image processing circuitry 208 required to perform the intensiveamount of processing required to create a digital image, a memory buffer262 to store the digital data as it is being assembled into a finisheddigital image, and finally image storage 264 to hold and maintain thefinal captured image 240.

The photosite array 209 consists of millions of tiny light cavities(“photosites”) which can be uncovered to collect and store the photonsof light reflected by an object or scene. Once the photosites havecollected photons, the camera closes each of the photosites and thendetermines how many photons were collected by each. The relativequantity of photons in each cavity are then sorted into variousintensity levels, whose precision is determined by bit depth (forexample, 0-255 for an 8-bit image, or any other appropriateimplementation).

However, the intensity levels calculated by the photosite array bythemselves would only create grayscale (black and white) images, sincethese photosite cavities are unable to distinguish how many photons ofeach color they received. In order to capture color values of something,a filter 260 has to be placed over each cavity that permits onlyparticular colors of light. A close-up view of one common type of filter260 is shown in FIG. 2A. Most current digital cameras can only captureone of three primary colors in each cavity, and so approximately ⅔ ofthe incoming light is captured by a photosite array 209 with a colorfilter 260 on the front.

As a result, a digital camera 220 has to approximate the other twoprimary colors in order to have full color at every photosite. A typicalway of doing this is to have the camera 220 look at the neighboringphotosites to see how much of the other color was received there, andthen interpolate a value for the current location. For instance, if aphotosite with a red filter only collects photons of red light, thenthat same photosite can look at the number of photons received by theneighboring or nearby blue photosites to determine the approximate bluevalue to use for the red photosite location. Something similar is donefor the green value at the photosite. In other words, in order to createan accurate image 240, steps must be taken to counteract the effectsintroduced by the filter 260.

The most common type of color filter is called a “Bayer array,” and thisarrangement of filter colors is shown in the close up of the filter 209shown in FIG. 2A. This arrangement has twice as many green filters as itdoes either red or blue. The Bayer array (and any other arrangement offilters) introduces a “mosaic” pattern to the light intensity valuescaptured in the photosite array 209, and so the “demosaicing process”step 207 is needed to create a final image 240 in order to get rid ofthe mosaic effect thus introduced.

The majority of the prior art inventions for grain quality sensing arebased on the analysis of final, capture images 240. This limits theseprior art inventions to accepting the “processing steps” (that is, steps206, 207, and 208, as well as other processes built into modern digitalcameras. Each of steps 206-208 may introduce changes in the creation ofthe final image 240 that ultimately must be “undone” during the grainquality determination process. In other words, prior art inventionswhich work by analyzing final captured images 240 are subject to theprocessing inherent in any modern digital camera or imaging device 220.

The present invention is an improvement in the art which “breaks open”the digital camera and looks at the raw photo data collected by thephotosite array 209 without creating a captured image 240. Turning toFIG. 2B, an optical data capture sensor 222 comprises a lens 204 forcapturing and directing photons of light into the photosite array 209.As with the imaging device 220 of the prior art, the photosite array 209of the optical data capture sensor 222 is covered with a filter 260 forcontrolling which frequencies of photons (light) are taken in by theindividual photosites in the photosite array 209.

The lens 204, filter 260, and photosite array 209 are the onlycomponents that the optical data capture sensor 222 has in common withthe imaging device 220 of the prior art. The optical data capture sensor222 does not do the same functions that are done by the analogelectronics 205, analog-to-digital converter 206, demosaicing process207, and digital imagine processing 208 of the prior art imaging device220. The optical data capture sensor 222 also does not require a buffer262 and image storage 264, as there is no final captured image 240created.

In place of the functions described in the previous paragraph, theoptical data capture sensor 222 uses the raw data collected by thephotosite array directly, without processing it and converting it into acaptured image 240. This is done in a series of array processingfunctions 210, which will be detailed in the discussion of FIGS. 3A-3C.

In an alternate embodiment of the present invention, the demosaicingprocess 207 may be added to the array processing functions 210 as ameans of increasing the performance of the grain quality analysis. Thiswill be explained in more detail in the discussion of FIGS. 3C and 3D.

The output of the array processing functions include information on thequality of the crop material 200, including the percentage of crackedgrain detected (222A), the percentage of material other than grain, orMOG (222B), and the percentage of clean grain (222C). The information222A, 222B, and 222C is calculated by the array processing functions 210without ever creating a final captured image 240.

Moving to FIG. 2C, we see a perspective view of the optical data capturesensor 222. While FIG. 2B was intended to detail the functional aspectsof one embodiment of an optical data capture sensor 222, FIG. 2C focusesmore on the physical implementation.

FIG. 2C again shows the lens 204, the filter 260, and the photositearray 209, as before. In addition to these components, light sources 211are added to FIG. 2C. These light sources 211 may be light emittingdiodes (LEDs) or any other appropriate lighting source. The number oflight sources 211 may vary from one embodiment to another, and thefrequency of light emitted by each light source 211 may be of adifferent wavelength, as may be required to capture the appropriatephoton data reflected back from the crop sample 200. The use of theselight sources 211 in analyzing the crop sample 200 will be discussedshortly.

A processing unit 212 provides power for the photosite array 209 andlight sources 211, controls the inputs and outputs from the optical datacapture sensor 222, and performs the processing carried out by the arrayprocessing functions 210. The entire module may be enclosed in an outerenclosure 214, shown here as a dotted line.

FIG. 2D is a side view of the embodiment of an optical data capturesensor 222 shown in FIG. 2C. It is provided to give an alternate view ofthe optical data capture sensor 222, but does not introduce any newfunctionality or components.

The following paragraphs shall describe one embodiment of an opticaldata capture sensor 222 and how it may be used to implement a grainquality sensor (also known as a “grain quality and cleanliness sensor”).The purpose of a grain quality sensor is to determine the levels ofmaterial other than grain (MOG) and broken kernels (cracked grain) inthe clean grain path. The values are reported to the operator andprovide inputs to the automation algorithm discussed later in thisspecification. The following description will refer to FIGS. 2B, 2C, and2D and will use the reference designators collectively from thesefigures as needed.

In one embodiment of a grain quality sensor, the crop sample 200 isilluminated with light sources 211 which emit, at a minimum, ultravioletlight (UV), green light, and red light. The wavelengths of the green andred light sources 211 are used to provide the maximum contrast among thecolor photosites in the photosite array 209. In other words, the greenlight source 211 should produce minimal excitation in the red and bluephotosites in the photosite array 209 (as dictated by the transmissioncurves of the color pattern filter 260).

Doing this will maximize the ability to perform coarse spectroscopy withthe 3 different types of photosites in the array 209 (those receivingonly green, those receiving only blue, and those receiving only redphotons). The UV light source 211 is chosen to provide maximum contrastbetween the reflective starchy interior of the grain and the bran, orouter casing, while maintaining reasonable sensitivity of the photositearray 209 and transmission through the lens 204 and filter 260.

A processing unit 212 analyses the raw photosite array 209 data anddetermines the fractional amount of MOG and cracked grain so that it canbe displayed to the operator and used as inputs to the automationalgorithm.

Basic Algorithm: By shining light of different wavelengths on the cropsample 200, information can be gathered by the grain quality sensor (bythe optical data capture sensor 222). Individual photosites from thearray 209 which are dark (indicating relatively few photons of lightcollected in those areas) may indicate voids in the sample or noise andcan be eliminated from consideration.

The inside of a grain kernel typically absorbs and reflects differentwavelengths of light than the outer casing of the kernel. This fact canbe used to detect damaged grain, as the wavelengths of light typicallyabsorbed by the cracked, exposed inner kernel will be different thanundamaged grain. The absorption and reflection of MOG will also bedifferent than the absorption and reflection of clean grain and damagedgrain.

The raw photosite array 209 data can then be analyzed for hue,saturation, and value (known as HSV by those skilled in the art, andalso sometimes known as HSI, for hue, saturation, and intensity) todetermine which photosites in the array 209 correspond to HSV valuesrepresenting cracked grain, clean grain, and MOG. This algorithm isexplained in detail in FIGS. 3A through 3D, and the correspondingdiscussion.

Variations on the Algorithm: Other color spaces can be used instead ofHSV, for example, using the ab plane of the Lab colorspace. Lightness orvalue (intensity of the black and white image) may also be useful inidentifying objects.

The image is broken into a series of sub-sections. Many of thesesections will contain only grain and the spread in the correspondingdistribution of values along any particular dimensions in whichevercolor space will be minimized. This minimum spread is used to determinethe thresholds for the entire image.

Notes: Hue is essentially the color of the light collected in thephotosite array. Saturation or chroma is a measure of the purity of thecolor, so that white or gray are at one extreme and red, orange oranother pure color are at the other extreme. Value is the lightness ofthe area, so that white and gray only vary in intensity.

FIG. 2E illustrates how the present invention can create “partialimages” or “layer images” that eliminate visual elements present in theoriginal subject matter. It is important to note at this point that thecreation of images as discussed here in FIG. 2E is not required for theoptical data capture sensor 222 previously discussed. As stated then,the optical data capture sensor 222 does NOT use captured images 240 todetermine information on the crop sample 200. This is a separatefunction which can be performed using the present invention.

The optical data capture sensor 222 can be used, as previouslydescribed, to detect which photosites in the array 209 containinformation related to clean grain 201, damaged grain 202, and/or MOG203.

It would be possible, therefore, to segment the photosites into one ofthese categories (clean grain, damaged grain, and MOG) and to then havean algorithm that will create “partial images” that do not faithfullyreproduce the original subject matter (in this case, the crop sample200), but instead show only subsets of the original sample 200. Forexample, one partial image 242A may show only the MOG 203 detected in asample. Other partial images (242B and 242C) show only the damaged grain202 (or just the damaged section of the grain kernels, 202A) or only theclean grain 201.

This “partial image” concept can be applied in areas other than grainquality sensing. For example, one can imagine a camera implementing thispresent invention (an alternate embodiment of the optical data capturesensor 222) which will eliminate certain color patterns from the finalproduced images, such as eliminating the blue sky from an outdoorpicture, and possibly replacing it with another color, such as white orblack.

FIG. 2F illustrates an alternate embodiment of a grain quality sensorthat detects damaged grain by detecting the lack of specular highlightson certain kernels. The previous discussion of a grain quality sensorand/or an optical data capture sensor may not apply well for all typesof seeds or kernels. For example, the inside of a soybean is essentiallythe same color as the outside casing of a soybean, so using the types ofcolor-based analysis of the raw pixel data as previously described maynot be effective. An alternate method of analysis may be required.

Turning to FIG. 2F, we see an assortment of whole soybeans 276 andbroken soybeans 278. When a light source 277 is shown on the soybeans276 and broken soybeans 278, we see that the outer casing of wholesoybeans 276 is “shiny” and will produce specular highlights 280 whichwill appear as bright spots in an image taken of the crop. On thecontrary, the inside surface 284 of a broken soybean 278 is not “shiny”and therefore does not produce a specular highlight 280. When an imageis taken of the soybeans (276, 278) with an imaging device 220 or anoptical data capture sensor 222, the image can be analyzed to look forthe number of soybeans 276 with specular highlights 280 and the numberor broken soybeans 278 without specular highlights 280.

It should be noted that, while the specification has previouslydiscussed grain quality sensors that do not use images or imageprocessing, standard image processing may be required to identify thespecular highlights on soybeans or other similar crops.

Turning now to FIG. 2G, we see an image that has been processed tohighlight “bright” spots or specular highlights. Each point in the image(or in the raw pixel data, if image processing is not used) is analyzedfor brightness/intensity. Those falling below a certain threshold areshown as black or dark points on the resulting image, and those meetingor above the threshold will be shown as light or white points on theimage. The result of this image processing will look like the imageshown in FIG. 2G. Please note that white arrows are used in FIG. 2G inplace of standard lead lines because of the dark nature of the imageshown.

The processed image as shown in FIG. 2G will show whole soybeans aswhite or light colored outlines 288 each containing a bright spotrepresenting a specular highlight 288A. Broken soybeans will show aswhite or light-colored outlines 286 without a specular highlight withinthe outline 286. Some broken soybeans may be shown as irregular shapes286A, indicating they are not whole soybeans.

An algorithm looking for broken soybeans in an image processed in thismanner could identify broken soybeans by looking for nonstandard shapes(such as 286A in FIG. 2G) or by looking for shapes that do not contain abright spot 288A within the outline (such as 286 in FIG. 2G).

It is important to note that this alternate embodiment of a grainquality sensor that uses the presence of specular highlights to identifyundamaged kernels or seeds will work with any crop type where theoutside casing of the kernel or seed is reflective and the insidesurface of the same type of kernel or seed is not. Soybeans are used inthe example but are not intended to be limiting in any way.

It is also important to note that the approach discussed above might beused to help identify material other than grain, or non-crop material.In a crop such as soybeans, the expected outline of the kernels will bea certain shape (in this case, generally circular) and a certain size.Any outlines outside of those expected shapes and sizes (for instance, arectangular shape for soybeans, or a size significantly larger than atypical soybean) are likely non-crop material. The presence of aspecular highlight inside of one of these “outlier outlines” would helpto identify the object as non-crop material, or to otherwise provideinformation on the nature of the object.

FIG. 3A is a flowchart showing the processes used to create the numericvalues and other outputs of the optical data capture sensor of thepresent invention. The steps shown in FIG. 3A represent one embodimentonly, and are not meant to be limiting, nor does the order of the stepsshown in the flow necessarily mean that the steps shown have to be donein a certain order. The key concept captured in FIG. 3A is that allarray processing functions 210 operate on individual photosite values(that is, on raw captured data) without producing a final image.

In Step 210A, each of the photosites in the photosite array 209 isanalyzed to determine the number of photons detected (indicative of theamount of light received) and a determination is made as to thewavelength of light represented by each photosite based on the filter260 that is covering the photosite array 209. In Step 210B, clusters ofsimilar color levels are identified, and each cluster is compared topredicted values for clean grain to determine which of these clustersrepresent clean grain (or what percentage of the overall photosites inthe array 209 appear to be representing clean grain). Steps 210C and210D do the same analysis to determine the overall percentage of bothMOG and damaged grain (or damaged crop), respectively. An optional Step210E is performed in order to eliminate outliers or photosites that donot appear to match any of the surrounding photosite values (in otherwords, a single “dark” photosite surrounded by photosites representingclean grain is eliminated as probable noise.)

Finally, in Step 210F, the determined percentages (material breakdownvalues) determined in Steps 210B, 210C, and 210D are sent to thecontroller responsible for making automated adjustments to the combineor for displaying the values to an operator.

In optional Step 210G, “partial images” such as those discussed andshown in FIG. 2E may be generated for display to an operator.

FIG. 3B illustrates one embodiment of an algorithm for analyzing valuesin a photosite array to determine the content of a grain or crop sample.Once raw data has been captured in the photosite array 209, all of thepixels in the array are grouped into subsets 300, and the “spread” ofeach pixel subset 300 is measured. In one embodiment, the “spread” isdetermined by taking the standard deviation of the pixels in the subset300. If the standard deviation of a particular subset 300 is small, thatmeans that all of the pixels in that subset 300 are close to the samecolor. A larger standard deviation in a subset 300 means that there is alarger “spread” or range of colors represented in the subset 300. Inthis example, which is meant purely for illustrative purposes, the sizeof the subset is 5 pixels, but any appropriate subset size may bechosen. Only a very small photosite array 209 is shown in FIG. 3B, justenough to illustrate the concept.

After the standard deviations for all of the subsets 300 have beendetermined, the subsets are placed in order by the size of the standarddeviation. For example, in the center of FIG. 3B, the subsets 300 areshown arranged vertically from the smallest standard deviation at thetop to the largest at the bottom. (This concept is shown graphically asa stack of subsets 300 in FIG. 3B, but the output in reality is a rankedlist of standard deviations, from smallest to largest.)

Once the ranked list is created, a small number of subsets 300 near thetop of the list (that is, the subsets 300 with the smallest standarddeviations) are considered to be the “reference distribution,” and theentire list of subsets 300 is considered to be the “total distribution.”

The subsets 300 in the “reference distribution” should ideally be thesubsets 300 that are closest to the desired color (for example, thecolor of clean grain). The histogram of the reference distribution canbe plotted against the histogram of the total distribution of colorscaptured by the photosite array.

This is shown on the right side of FIG. 3B. In the plot of thehistograms 305, the smaller dashed line represents the plot of thereference distribution 310. In this example, the histograms are plottedon a three-dimensional plot in the HSV color space (representing Hue,Saturation, and Value), but any other appropriate color space can beused with similar results.

It should be noted that the plot 305 is shown in only two dimensions(hue on the Y axis and saturation on the X axis), but there would alsobe a third axis rising up out of the figure, perpendicular to both the Xand Y axes, and that would represent intensity. The intensity has beenomitted for clarity in FIG. 3B, but its effect would be to give thehistogram plots of the reference distribution 310 and the totaldistribution 315 a volume, with a third dimension of the two plotsrising up out of the figure in the direction of the missing intensityaxis.

The total distribution plot 315 is added to the histogram plot 305,superimposing it on the reference distribution plot 310. The totaldistribution plot 315 will always be at least as big as the referencedistribution plot 310, but will typically be significantly larger,representing the larger color range present in the total distributionover the reference distribution. If the grain quality sensor is lookingat a very pure sample of grain (that is, a sample that is almost 100percent clean grain), the total distribution plot 315 may be almost assmall as the reference distribution plot 310.

In one embodiment, the algorithm illustrated in FIG. 3B looks for thepoint of peak intensity 325 in the reference distribution plot 310.Although the intensity axis has been intentionally left off of the plot305 shown here, the point of peak intensity 325 would be the point atwhich the reference distribution plot 310 extends the farthest into theintensity dimension (the tallest peak that would extend up out of thefigure if intensity were plotted.

This point of peak intensity 325 is used to draw a separation line 320on the graph perpendicular to the hue axis (it would be a plane if drawnin three dimensions). This line is used to determine relativepercentages of clean grain, MOG, and cracked or broken grain in thefollowing way:

-   -   A point inside the reference distribution plot 310 will be        considered to represent clean grain.    -   A point outside of the reference distribution plot 310 and ABOVE        the separation line 320 will be considered to represent MOG.    -   A point outside of the reference distribution plot 310 and BELOW        the separation line 320 will be considered to represent cracked        or broken grain.

The above bullets assume that the hues are plotted such that the colorsrepresenting MOG will be more likely found toward the top of thetwo-dimensional plot, and colors representing broken grain will betoward the bottom. The spectrum of hues could be plotted in reverse, andthen the sides of the separation line 320 representing MOG and brokengrain would be flipped.

In the method outlined above, the data captured in the photosite array209 can be analyzed without ever forming an actual image. Stated anotherway, to create an image from the data captured by photosite array 209the spatial information (that is, the location of each pixel in relationto every other pixel in the array 209, or its X-Y location in the array209) must be maintained so that the data makes sense as an image.However, the algorithm described here and in FIG. 3B is only looking atthe total distribution of colors in the photosite array 209, withoutcaring about the locations in the photosite array 209 that held the dataoriginally.

An analogy may help better illustrate this concept. Let's imagine thatan “image” is the picture printed on a fully assembled jigsaw puzzle,and the unassembled pieces of the puzzle scattered over an area on atable represent the photons captured by the photosite array. In orderfor an “image-based” grain quality algorithm from the prior art to work,the “jigsaw puzzle” must first be completely assembled (representing thecreation of an image) before those algorithms can work.

However, the algorithm illustrated in FIG. 3B does not care about the“image” on the assembled jigsaw puzzle; it only cares about the datarepresented by the individual puzzle pieces. The subsets 300 (from FIG.3B) do not have to be created from pixels or elements from consecutivelocations in the photosite array 209. The algorithm of the presentinvention would work if someone randomly picked puzzle pieces from thebox (representing random elements in the photosite array) to form eachsubset, and the actual puzzle never has to be assembled (that is, noimage ever has to be created).

Even though the demosaicing process previously discussed in thisspecification does not have to be used in the algorithm of the presentinvention, since no image needs to be created, it can be applied to thedata in the photosite array 209 to achieve improved results, as isdescribed briefly in FIGS. 3C and 3D.

FIG. 3C describes how the demosaicing process of the prior art works invery high-level terms. It has been discussed previously in thisspecification that a color filter is placed on top of the photositearray 209 so that each individual “bucket” or element in the photositearray 209 will only capture one color of photon (either red, green, orblue). Turning to FIG. 3C, we focus on a single pixel 335 taken from theupper left corner of the photosite array 209 to describe the process.Each pixel 335 on a photosite array 209 is represented by four elementsin the photosite array 209, one red-filtered element 335R, twogreen-filtered elements 335G1 and 335G2, and one blue-filtered element335B. For discussion purposes, we will focus on the 335G1 element inFIG. 3C. When the photosite array 209 is used to capture raw photons, itis very likely that photons of red, green, and blue light will allstrike the 335G1 element, but, because of the green filter over 335G1,only the green photons will be allowed to actually enter the 335G1element. Because of this filtering, the 335G1 element will likely bemissing detail that is in the point in the physical world correspondingto element 335G1 (that is, the blue and red photons that may have hitthe element but been rejected and not counted).

The demosaicing process from the prior art can be used to correct thisdeficiency. In order to determine the amount of red photons that mayhave hit the 335G1 element and been rejected, an algorithm can look atthe closest red-filtered elements and estimate the amount of red thatmay have hit 335G1 based on the number of red photons the closestred-filtered elements received.

For example, for element 335G1, an algorithm may look at thered-filtered elements 335R, 335N1, 335N2, and 335N3 to see how many redphotons they captured. The red-filtered elements closest to 335G1 (suchas 335R and 335N2) will have a greater effect on the calculated redvalue for 335G1 than those red-filtered elements farther away (such as335N1 and 335N3). By looking at the closest red-filtered neighbors, anestimated value for the number of red photons that were likely receivedat element 335G1 is calculated. This new value is put into a new“red-value array” 336 as value R_(G1), in the location corresponding tothe 335G1 element in the original photosite array 209.

Using this method, the demosaicing process will create a new red-valuearray 336 the same size as the original photosite array 209, as well asa green-value array 337, and a blue-value array 338. The result of thisprocess is that there is now three times as much information(represented by the three arrays 336, 337, and 338) than was captured inthe original photosite array. This increase in data can improve theresults achieved by the grain quality sensor of the present invention.

FIG. 3D does not introduce any new concepts, but shows the results ofthe process from a higher level of abstraction, showing that the datacaptured originally in the photosite array 209 is multiplied in theprocess, outputting arrays 336, 337, and 338, one array corresponding toeach of the three colors.

FIG. 4A shows the clean grain elevator of a typical combine and thesensors associated with the clean grain elevator 400 as defined for usein the present invention. The clean grain elevator 400 in a combineprovides a mechanism for delivering the collected (harvested) grain fromthe clean grain auger (114, FIG. 1) to the clean grain tank 110. Notethat the clean grain auger 114 is partially obscured in FIG. 4A (as itwould be behind the bottom of the clean grain elevator 400), but itdelivers the “clean grain” collected at the bottom of the cleaning shoe117 to the clean grain tank 110. Refer to FIG. 1 and FIG. 4A to identifyall of the parts referenced herein.

Returning to FIG. 4A, we see that clean grain (or simply “grain”) 405 isdelivered into the bottom of the clean grain elevator 400 by the cleangrain auger 114. Paddles 403 mounted on a delivery conveyor 404 rotateup through the clean grain elevator 400 to deliver the grain 405 to theclean grain tank 110. (Note: In the example shown in FIG. 4A, theconveyor 404 and paddles 403 rotate in a clockwise manner.)

The optical data capture sensor 222 will be mounted such that it canexamine the grain 405 before it is deposited in the grain tank 110.There are several methods of mounting the optical data capture sensor222 to the clean grain elevator 400, and one possible embodiment of sucha mounting method is shown in FIG. 4A. In this mounting method, anopening 406 is made in the side of the clean grain elevator 400 suchthat some of the grain 405 spills into a viewing chamber 409 which ismounted on the clean grain elevator 400. The grain 405 travels throughthe viewing chamber 409 and is “presented” to the optical data capturesensor 222. The optical data capture sensor 222 is mounted to theviewing chamber 409 such that the lens 204 of the optical data capturesensor (see FIG. 2B) is focused on the contents of the viewing chamber409. The optical data capture sensor 222 is activated to illuminate thegrain 405 held in the viewing chamber 409 and capture photons reflectedfrom the grain 405 using the photosite array (209, see FIG. 2C). Oncethe data is collected form the grain 405, a return auger 408 takes thesampled grain 405 and deposits it back into the clean grain elevator 400so that it can continue its journey into the clean grain tank 110.

It should be noted that this method of mounting the optical data capturesensor 222 to the clean grain elevator 400 is only one embodiment, andother means of mounting the optical data capture sensor 222 to the cleangrain elevator 400 do exist and may be used in place of the method shownin FIG. 4.

FIG. 4B shows an alternate mounting location and system for the opticaldata capture sensor 222 (such as a grain quality sensor) of the presentinvention. The upper portion of a clean grain elevator 400 is shown,showing the paddles 403, grain 405, clean grain tank 110, and unloadingauger 402. The yield sensor 401 and moisture sensor 122 shown in FIG. 4Ahave been removed in FIG. 4B for clarity. In this alternate mountingscheme, the optical data capture sensor 222 is mounted at the top of theclean grain elevator 400, right above the point where the grain 405 isthrown off of the paddles 403 into the clean grain tank 110.

In this location, the optical data capture sensor 222 does not need tohave the viewing chamber 409 or the return auger 408, as the flow ofgrain 405 is not interrupted (no sampling from the elevator 400) isrequired. Instead, in this location, the optical data capture sensor 222will capture raw photon data as the grain 405 flies past the opticaldata capture sensor 220. By capturing photon data as the grain 405 is inflight, a better representation of the grain 405 may be obtained, as itis not packed into a tight viewing chamber 409.

IV. Yield Sensor and Moisture Sensor

Returning now to look at FIG. 4A, at the top point of the conveyor 404,the grain 405 is thrown into the tank 110 by centrifugal force as thepaddles 403 switch directions and begin the descent back down the cleangrain elevator 400. Inside the clean grain tank 110, there areoptionally two additional sensors provided to collect data for use onthe combine.

A yield sensor 401 is placed in the path of the grain 405 that isejected from the paddles 403. Grain 405 strikes the yield sensor 401 andthe yield sensor 401 calculates the amount of grain 405 striking it andcalculates the approximate yield (amount of clean grain) entering thetank at any given moment.

The yield sensor 401 may be implemented by a variety of methods. Onecommon method in used today is to have the grain 405 hit an impact plateattached to a load sensor. The force of the grain 405 hitting the impactplate allows the approximate load to be measured and allowing aderivation of approximate mass or material flow rate.

Another means of creating a yield sensor is to base the sensor on anacoustic chamber such as that used by the energy sensing acoustictechnology (ESAT) sensors manufactured by Appareo Systems, LLC of Fargo,N.D., including those disclosed in WO/2012/125575, the publication ofwhich is incorporated herein by reference in its entirety, including thesystem and method for determining yield and/or loss from a harvestingmachine using acoustic sensors, as disclosed in US/2014/0135082, thepublication of which is incorporated herein by reference in itsentirety, or variants thereof of any of the above described acousticsensor technologies. An acoustic sensor such as those described in thereferenced documents determines the amount of the yield based on theamount of sound generated by an impact on an impact plate sitting atopan acoustic chamber.

Yet another alternate method of determining the yield would be to placeload sensors on the upper bearings of the conveyor 404 in the cleangrain elevator 400. The load on the sensors could be taken when theclean grain elevator 400 is empty, and then compared to the load on theconveyor 404 when material is flowing through the clean grain elevator400. The load value when the clean grain elevator 400 is empty could bemeasured once during a configuration step (perhaps as part of a factoryconfiguration step) and stored in non-volatile memory for subsequentcomparison to the load value when crop is present. The differencebetween the two readings would represent the mass of the clean grain(and hence give the yield).

Any other appropriate method of determining yield may be used withoutdeviating from the intent of the present invention.

In addition to a yield sensor, a moisture sensor 122 may also be placedinside the clean grain tank 110. There are various ways to implement amoisture sensor 122 available in the art. One such common type ofmoisture sensor 122 is a capacitive sensor. A capacitive sensor measuresmoisture by monitoring the change in the dielectric properties of grain.Another common type of moisture sensor 122 uses near-infrared (NIR)wavelengths of light to detect moisture. This is done by shining twodifferent wavelengths of NIR on a sample. One of the wavelengths iscalibrated for moisture and the other as a reference. The ratio of thetwo signals is derived electronically to calculate the moisture content.The convoluted nature of NIR spectra can require broadband illumination,a spectrometer, and chemo-metric calibration methods to accuratelyextract moisture. Often the moisture sensor 122 collects samples of crop405 in the clean grain tank 110 in a funnel-shaped cup, performs theanalysis, and then releases the crop 405 such that it drops to thebottom of the tank 110 and can be offloaded subsequently by an unloadingauger 402.

One improvement on the prior art use of NIR moisture measurement is theused of two or more MEMS spectrometers. MEMS spectrometers are smallerand less expensive than traditional spectrometers, making them perfectlysuited for such applications. When at least two spectrometers are used,one could be used to measure the crop sample and the other could be usedto measure the light source itself. The spectroscopic measurement of thelight source can be used as the “standard” or control data against whichthe spectroscopic measurement of the crop sample is compared, allowingfor highly accurate measurements that are free from environmentalvariations.

FIG. 4A illustrates one embodiment of an optical data capture sensor222, a yield sensor 401, and a moisture sensor 122. The exampleillustration in FIG. 4A is not intended to be limiting, and otherembodiments of the sensors can be created without deviating from theinventive concept captured herein. These sensors provide data itemswhich may be used independently (perhaps displayed to an operator), orin some combination in a combine control algorithm.

V. Look-Ahead and Look-Aside Crop Mass Sensor

FIGS. 5 through 6J detail one or more embodiments and components of amass load detection sensor (also called a “crop mass sensor”) which canbe used to predict the amount of crop material that is about to enterthe combine at any given moment in time. There are two main embodimentsof a crop mass sensor discussed herein: a “look-ahead” crop mass sensorwhich senses mass immediately in front of the combine, just before itenters the combine, and a “look-aside” crop mass sensor, which functionsessentially identically to the look-ahead sensor but focuses sensors tothe side of the combine instead of just in front of it, where theimproved viewing angles for sensors (the ability to look straight downon the crop from above, versus trying to detect mass by looking outahead of the vehicle) can give improved performance. Sensing mass to theside of the combine instead of directly in front of the combine alsomeans that the mass data being calculated will have to be stored andused on the subsequent pass of the vehicle (instead of being used almostimmediately with the look-ahead embodiment). Unless otherwise specified,the functional description below will apply to both the “look-head”sensor and the “look-aside” sensor, even if only one type of the twosensors is discussed.

FIG. 5 shows the main functional components of one embodiment of alook-ahead sensor of the present invention. A subset of the componentsshown here may be used to detect crop mass.

Several sensing technologies may be used separately or in combination todetect crop mass. These technologies are shown in FIG. 5. The sensorsdescribed will be mounted to a combine 500 and in one embodiment may bemounted to the top of the combine cab 501, as shown in FIG. 5, althoughother mounting locations are possible.

A look-ahead sensor 506 is shown in FIG. 5 as a collection of sensors ina common housing, shown as a dashed line. Other embodiments of thelook-ahead sensor 506 may exist which do not have a common housing, andwhich may have only a subset of the sensor technologies shown here.

In the embodiment of the look-ahead sensor 506 shown here, thelook-ahead sensor 506 comprises a imaging device 502, a LIDAR sensor503, and two radar sensors, one radar at a frequency that is absorbed bywater 504 and one radar at a frequency that will pass through the cropto detect the ground beyond or beneath the crop 505. Each of thesecomponents shall be described separately in the following paragraphs.

VI. Imaging Device 502

A visible-spectrum, high-resolution camera or imaging device 502 willrecord video footage of the combine harvesting the crop. Imagineprocessing algorithms will be used to analyze the captured images andvideo to help provide data that can be used to determine crop mass.

The type of image processing algorithm used may be dependent on the typeof crop being analyzed. For example, a flood fill algorithm could beused for wheat to look for areas of similar texture and/or color. Moreadvanced algorithms can be used to more accurately determine cropdensity. One possible implementation of the imaging device 502 that iscommercially available is a Basler Ace acA19t20-25gc camera with a 6 mmlens, although any appropriate imaging device could be used.

VII. LIDAR Sensor 503

A LIDAR system or LIDAR sensor 503 will also be used to help determinecrop mass in some embodiments of the crop mass sensor 506. A 2D/3D LIDAR503 works by firing pulses of laser light at a target and determiningthe distance to the target by measuring the time it takes for the lightfrom the laser to be reflected back to the LIDAR 503 sensor.

By moving the LIDAR 503 forward (that is, by moving the combine 500forward, thereby effectively moving the LIDAR 503 forward deeper intothe crop) and constantly taking measurements, a three-dimensional modelof the can be constructed layer by layer as the LIDAR 503 takes newreadings on the distance of the continually changing front edge of thecrop.

When using a LIDAR system 503 during harvesting, some of the laserpulses will not hit the crop, passing through to the ground. Theremaining pulses will hit the crop and reflect back. The ratio of pulsesthat hit the ground to pulses that hit the crop helps to determine cropthickness. One possible embodiment of the LIDAR sensor 503 that iscommercially available is a Hokuyo UTM-30LX-EW, although any appropriateLIDAR sensor or similar technology could be used.

VIII. Radar

The radar system will use two distinct radar bands. The frequency bandof the moisture-detecting radar 504 will be such that it is stronglyabsorbed by moisture (and therefore crop material which has a measurablewater content), and the non-moisture detecting radar 505 will be weaklyabsorbed by water and thus will pass through to the ground. The ratiobetween or distance between absorbed energy (from radar 504) andreflected energy (from radar 506) will be used to help correlate thecrop density.

An example product that might be used for the moisture-detecting radar504 is Delphi RSDS 77 GHz radar, although any appropriate type of radarcapable of being absorbed by moisture could be used.

An example product that might be used for the non-moisture-detectingradar 505 is a 24 GHz radar system from Delta Mobile, although anyappropriate type of radar capable of being passed through moisture couldbe used.

FIG. 6A shows a top view of a combine showing how the radar-basedcomponents of the look-ahead sensor of FIG. 5 would work to help predictincoming crop load. The look-ahead sensor 506 will emit two separatefrequencies of radar energy, one that is strongly absorbed by moisture601 and one that will pass through the crop 602. The differences betweenthe moisture-absorbing band 601 and the non-moisture-absorbing band 602can be used to help calculate the amount of crop matter 610 is presentin proximity to the combine 500, and which is about to enter the combineheader 523 to be cut and pulled into the machine. The arrow 651indicates the forward direction of travel for the combine 500.

FIG. 6B shows a top view of a combine showing how the LIDAR-basedcomponent of the look-ahead sensor of FIG. 5 would work to predictincoming crop load. The look-ahead sensor 506 will emit beams of focusedlight 603 that will either strike crop matter 610 and be reflected backor will miss the crop matter 610 and not be reflected back, or have areflection that is significantly delayed and or reduced by bouncing offthe ground instead of the closer crop material. The differences betweenthe light sensed reflecting back from the crop matter and that not beingreflected back can be used to help calculate the amount of crop matter610 is present in proximity to the combine 500, and which is about toenter the combine header 523 to be cut and pulled into the machine. Thearrow 651 indicates the forward direction of travel for the combine 500.

FIG. 6C shows a top view of a combine using an alternate embodiment ofthe look-ahead sensor of the present invention which looks to the sideof the combine, instead of ahead of the combine. In this embodiment ofthe crop mass sensor 506, the crop mass sensor 506 is focused to theside of the combine 500 instead of to the front of the combine 500. InFIG. 6C, the broadcast energy 670 is meant to represent all of thedifferent sensing technologies that may be present in the crop masssensor 506, including, in some embodiments, the video sensing, LIDARlight energy, and radar frequencies previously discussed in thisspecification, or some subset thereof (or potentially with additionaltechnologies not discussed herein).

By focusing the “look-aside” sensor (functionally equivalent to thelook-ahead sensor or, more generically, the crop mass sensor, and thusshown using the same reference number 506) to the side of the combine500 instead of in front of the combine 500, the look-aside sensor 506has an improved angle for sensing crop mass, as the broadcast energy 670can be projected down on the crop material 610 at a steeper, morevertical angle, allowing better detection of crop material 610 versustrying to look out ahead of the combine 500.

This may require that two look-aside sensors 506 be mounted on thecombine 500, such that the mass can be detected on either side of thecombine 500 depending on the direction the combine 500 is traveling.Alternately, one look-aside sensor 506 could be used but somehow movedfrom one side of the combine 500 to the other, either by hand before thecrop is harvested or automatically with a positioning system.Alternately, the look-aside sensor 506 could be placed on just one sideof the combine 500 permanently, requiring the operator to always movethrough the field in such that the permanently-mounted sensor 506 isalways facing the subsequent pass in the field.

Because the look-aside sensor 506 is looking to the side of the combine500 (that is, at the crop mass 610 to one side or other of the combine500), the first pass through the field will not have any stored cropmass data to rely on.

It is important to note that one major difference in the processing fora look-aside version of the sensor 506 versus the look-head version isthat the crop mass detected at any given time must be stored for lateruse, along with a location for which the stored data applies. That is,the data collected on the first pass (or the current pass) will need tocontain some kind of location such that the data can be used at theappropriate point of travel on the subsequent pass. It is also possiblethat the crop mass reading from the look-aside sensor can be saved andreused by the machine at a future time, should harvesting beinterrupted.

Another important note about the look-aside sensor is that, as it is notsensing an area of crop that is immediately going to enter the combinedoing the sensing, then the crop mass information can be transmitted toother machines working in the same field. Jumping ahead in the figuresto FIG. 6J, this figure illustrates the concept. In this examplescenario, three separate combines (labeled 500A, 500B, and 500C todistinguish them in the figure, but identical otherwise in function tocombine 500 on other drawings) are harvesting in a field together. Thisis a common scenario for contract harvesting companies that travel fromfield to field and harvest fields as a pay service for farmers. AlthoughFIG. 6J shows the combines traveling in the field in the same direction,slightly staggered, other relationships in placement and direction oftravel are possible without deviating from the intent of the invention.

As combine 500A travels through the field, harvesting plants 610, isuses its look-aside sensor 506 to sense the plants 610 in the next swathover from its current position. This information is then transmitted viaa wireless communications link 688 to combine 500B, so that combine 500Bcan see the mass that it will be coming into. Combine 500B does the samefor combine 500C.

It should be noted that the crop mass information may be transmitted toall harvesting machines on the field, and not necessarily to onespecific machine. If one of the machines is taken out of service, thenall machines have the same crop mass data, which also contains locationdata. Whichever combine machine gets to that “sensed area” first willuse the crop mass data thus received to configure the combineaccordingly, or to report to the operator for their information.

FIGS. 6D through 6J illustrate an alternate embodiment of the LIDARportion of the crop mass sensor 506. FIG. 6D illustrates how ahorizontal line of light 621D emitted by a laser device 620 appears tobe straight when displayed on a flat wall surface. The wall shapeassumed in FIG. 6D is shown in a top view 621A (showing the wall'sprofile), and a front view 621B (showing how the horizontal line oflight will be seen when viewed from the front on a flat wall. Theperceived line 621C when seen from the front view is a straighthorizontal line profile.

FIG. 6E illustrates how a horizontal line of light 622D emitted by alaser device 620 appears to be “broken” or displayed in “stair steps”when displayed on an uneven wall surface. The wall shape assumed in FIG.6E is shown in a top view 622A (showing the wall's profile, which hasvarying thickness or depth depending on which portion of the wall youare looking at), and a front view 622B (showing how the horizontal lineof light will be seen when viewed from the front on an uneven wall. Theperceived line 622C when seen from the front view is a line consistingof a series of steps, where portions of the displayed line 622D hit asection of wall that is closer to the laser 620 versus how the steps aredisplayed when the line 622D is displayed on sections of wall that arefarther away.

This concept may be better understood by looking at FIGS. 6F and 6G.FIG. 6F shows the side view 623 of the flat wall section of FIG. 6D.From this view, it becomes apparent that the laser 620 should shine onthe wall from an angle that is not perpendicular to the wall, and thatthe result should be viewed from an angle that is more or lessperpendicular to the wall. An imaging device 625, such as a camera, isplaced at an angle that is substantially perpendicular to the wall. Aline will appear on wall 623 at a point 626 where the light beam 675strikes the wall.

FIG. 6G shows the side view 624 of the uneven wall section of FIG. 6E.When an uneven wall 624 is used, the emitted light beam 675 will strikethe wall section 624 sooner for sections of wall 627 that are closer tothe laser 620, and later for sections of wall 628 that are farther backfrom the laser 620. When an imaging device 625, such as a camera, placedat an angle that is substantially perpendicular to the wall, views theline, segments of the line will appear higher when displayed on sectionsof wall that are closer 627, and other segments that are farther back628 will display the line in a position that is relatively lower.

FIG. 6H shows how the “structured light” concept for detecting theuneven surface of a wall (as illustrated in FIGS. 6D through 6G) can beextended to detecting crop mass in an agricultural situation. If thelaser 620 is mounted higher up on the combine 500 and projects ahorizontal line on the front “wall” of the crop material 610 that it isapproaching, the perceived line when seen from an imaging device 625that is mounted in a position such that it perceives the line from anangle that is approximately perpendicular to the direction of travel ofthe combine 500 will appear to be distorted, with some sections of theperceived line being higher than others.

Looking at FIG. 6I, we see one example of a perceived line 635 thatmight appear in an image from an imaging device 625 perceiving the lineas described above. The line will appear to be higher in spots where thehorizontal line is displayed on crop material 610 that is closer tolaser 620 and lower in spots where the line is displayed on cropmaterial 610 that is farther away from the laser 620.

For example, location 635B appears to be the lowest point of perceivedline 635, indicating that this spot corresponds to the point on the cropmaterial 610 that is farthest from the laser 620. Similarly, location635A appears to be the highest point of perceived line 635, indicatingthat this spot corresponds to the point on the crop material 610 that isclosest to the laser 620. A break or gap 635C in the perceived line 635likely indicates an area where there was no crop material 610 at all, orwhere the crop material 610 was too far from the combine to be detected,since there would be no surface onto which the perceived line 635 couldbe displayed.

The shape of perceived line 635 can thus be used to gather data on theshape of the front wall of the mass of crop material 610 as a combine500 moves through a field, and this shape information can be used tocreate a three-dimensional model of the crop mass before it is pulledinto the combine 500 itself.

IX. Mobile Device User Interface

FIGS. 7A-7C show aspects of one embodiment of an application userinterface for the present invention as displayed on a mobile computingdevice. In at least one embodiment, the combine automation system of thepresent invention has the ability to communicate wirelessly withexternal devices, which may include mobile devices such as smart phones,tablet computers (such as the iPad by Apple), laptops, other vehicles,and any other appropriate mobile device. In at least one embodiment, thecombine automation system of the present invention uses a mobile deviceas a display and user interface.

Turning to FIG. 7A, we see one embodiment of an application userinterface for the present invention as displayed on a mobile computingdevice 700. The mobile device 700 has a display screen 702 which may beused as a system display, showing the status of the system and theresults gathered by or calculated from the system sensors previouslydescribed in this specification. In this example page in FIG. 7A, agraphical representation 706 of the combine is displayed, and importantvalues such as sensor readings 704 are displayed superimposed orproximal to the graphical representation 706. Since display 702 is acomputer display, the actual readings and types of graphics displayedare virtually unlimited, but as shown in FIG. 7A typical sensor values704 may include the percentage of damaged (cracked) grain, thepercentage of MOG, the moisture content of the grain, the grain yield,the combine speed and engine RPMs, settings of the cleaning shoe andother combine subsystems, and productivity information (such as acresper hour). The display 702 can be used to send system messages to theoperator. These messages may include directives such as a recommendationto increase or decrease speed depending on the sensed condition of theharvested crop.

FIG. 7B shows another embodiment of an application user interface pagefor the present invention as displayed on a mobile computing device 700.In this example page, a “pop up” window 708 is displayed on the displayscreen 702. This pop up window 708 may include detailed information on acombine subsystem or may allow access to a user control. In the exampleshown in FIG. 7B, the pop up window 708 shows a system control 714 whichallows the operator in select the best operating mode for the combine.The circle control shown on system control 714 can be moved over thetriangular shape by the operator to command that the system focus moreon certain harvesting parameters than others. Once the proper set pointis selected, the operator can commit that by pressing the “set” key 710.Once the desired attributes are selected, the algorithms controllingautomatic combine adjustment will use this information to determine howto set the combine's system parameters.

FIG. 7C shows yet another embodiment of an application user interfacepage for the present invention as displayed on a mobile computing device700. In this example, a subwindow 712 showing the images of crackedgrain is displayed in the main window 702.

All of the example pages shown in FIGS. 7A through 7C are examples onlyand not meant to be limiting in any way.

X. Control System and Algorithms

FIG. 8 shows a series of combine adjustments that may be made by thepresent invention, as well as the system inputs that are used todetermine what adjustments are to be made. Focusing first on the rightmost column of the figure, we see a list of combine adjustments 950 thatcan be made and which affect the quality and/or quantity of the grainsuccessfully harvested.

The combine adjustments 950 are the system parameters that can bechanged to try to find the optimal operating efficiency of a combine,and they comprise the ground speed 822, concave setting 824, rotor speed826, fan speed 828, chaffer opening 830, and sieve opening 832.

Each of these adjustments 950 may have an effect on the operationalefficiency of the combine:

-   -   If the ground speed 822 is too fast, then it is possible that        the plant material being pulled into the combine will overload        the machine and cause a jam; it the ground speed 822 is too slow        then the machine may be underused.    -   If the concave spacing 824 is too close, movement against the        rotor may cause damage to the grain; if the concave spacing 824        is too far, then the grain may not be fully threshed.    -   If the rotor speed 826 is too fast, the plant material may bind        up and overload the rotor; if the rotor speed 826 is too slow,        proper threshing may not occur.    -   If the fan speed 828 is too fast, then the air stream it        generates may blow clean grain out the back along with the        lighter chaff and MOG; if the fan speed 828 is too slow, then        the air may not be strong enough to sufficiently lift MOG out of        the clean grain.    -   If the chaffer opening 830 is too open, then bits of MOG may        fall through along with the clean grain; if the chaffer opening        830 is too closed, the clean grain may not pass through.    -   If the sieve opening 832 is too open, then bits of MOG may fall        through along with the clean grain; if the sieve opening 832 is        too closed, the clean grain may not pass through.

The if-then statements provided in the bullets immediately precedingthis paragraph are provided as examples of behavior that may be seen insome embodiments of the present invention, and they are not meant to belimiting. Other relationships between system inputs 900 and combineadjustments 950 may exist in other embodiments of the present invention.There may also be other system inputs 900, or some of those systeminputs 900 presented herein may be removed or altered, in otherembodiments of the present invention. The same applies to the combineadjustments 950. The combine system represented in these examples is onepossible embodiment, and alternate embodiments of this architecture mayexist without deviating from the present invention.

The combine control system must be able to determine when each of thesecombine adjustments 950 is improperly set without human intervention inorder for the automation of the combine to be realized. In order to dothis, the combine control system will look at various combinations ofthe system inputs 900 to determine which combine adjustments 950 areimproperly set. Arrows are drawn from each system input 900 out to eachof the combine adjustments 950 that they correspond to.

For example, the following system inputs 900 are used, individually orin combination, to determine of the ground speed 822 is too low or toohigh:

-   -   Look Ahead Sensor 800    -   Feeder Housing Load 802    -   Rotor Load 804    -   Engine Load 806    -   Processor Material Distribution 808    -   Grain Loss 810    -   Chaffer Material Distribution 812    -   Clean Grain/MOG % 814    -   Cracked Grain %    -   Return Tailings 818    -   Crop Moisture 820

The values taken from these 5 system inputs 900 help the combineautomation system determine if the ground speed 822 needs to beadjusted. If the look ahead sensor 800 shows that a large mass of cropis about to enter the machine, than the combine automation system mayrecommend that the ground speed 822 be lowered so that the combine canhandle the increased load. All of the system inputs 900 that are used incalculating the appropriate ground speed setting 822 are load based.That is, they all provide information on the load the machine is eithercurrently managing, or is about to. If there is too much mass or load onthe system, the ground speed 822 needs to be lowered.

The other combine adjustments 950 are determined in a similar fashion.

FIG. 9 shows one embodiment of a control system architecture for thepresent invention. The control system consists of three tiers. At thetop tier 910, the system inputs 900 are filtered with a fast low-passfilter. The control outputs are evaluated once per second and controlthe ground speed 822 and fan speed 828 combine adjustments 950. Theseoutputs are sent to the combine automation system 940, which uses theinformation from the outputs to change the appropriate combineadjustments 950.

The next tier 920 will use a slower low-pass filter on the system inputs900. The control outputs are evaluated once per minute, and control therotor speed 826, chaffer opening 830, and sieve opening 832.

The last tier 930 will use a very slow low-pass filter on the systeminputs 900. The control outputs are evaluated once every 15 minutes andcontrol the concave spacing 824.

FIGS. 10A through 10N are a series of flowcharts which capture logicthat may be used by the present invention to determine which combineadjustments to make. It is important to note that FIGS. 10A through 10Nare provided as examples of logic that may be used in one or moreembodiments of the present invention, but they are not meant to belimiting in any way. Other logic arrangements may exist and may be usedin other embodiments without deviating from the inventive concept of thepresent invention.

FIG. 10A shows four conditions which can be used individually or incombination to determine if the ground speed is too fast. Theseconditions are the material distribution being toward the back of thecombine, high power and high fuel usage, high rotor torque, and highfeeder load.

FIG. 10B shows four conditions which can be used individually or incombination to determine if the ground speed is too slow. Theseconditions are the material distribution being toward the front of thecombine, low power and low fuel usage, low rotor torque, and low feederload.

FIG. 10C shows six conditions which can be used individually or incombination to determine if the concave is too closed. These conditionsare the material distribution being toward the front of the combine, ahigh amount of cracked or damaged grain, a low moisture content in thecrop (dry crop), high power and high fuel usage, high rotor torque, andan increasing level of MOG in the grain.

FIG. 10D shows three conditions which can be used individually or incombination to determine if the concave is too open. These conditionsare a light combine load, a high moisture content in the crop (wetcrop), and material distribution being shifted too far back.

FIG. 10E shows one condition which can be used to determine if the rotorspeed should be minimized, and this condition is a low moisture contentin the crop.

FIG. 10F shows one condition which can be used to determine if the rotorspeed should be maximized, and this condition is a high moisture contentin the crop.

FIG. 10G shows one condition which can be used to determine if the rotorspeed should be decreased, and this condition is a high percentage ofcracked or damaged grain.

FIG. 10H shows two conditions which can be used to determine if therotor speed should be increased, and these conditions are materialdistribution shifted to the back and a high processor loss.

FIG. 10I shows two conditions which can be used to determine if the fanspeed should be increased, and these conditions are a high percentage ofMOG seen at the chaffer and a high amount of returns.

FIG. 10J shows two conditions which can be used to determine if the fanspeed should be decreased, and these conditions are a high loss seen atthe chaffer and chaffer distribution is shifted toward the back.

FIG. 10K shows three conditions which can be used to determine if thechaffer opening should be closed down, and these conditions are a highpercentage of MOG seen at the hopper, a high amount of returns, and ahigh percentage of MOG seen at the chaffer.

FIG. 10L shows one condition which can be used to determine if thechaffer opening should be opened up, and this condition is a highsloughing loss.

FIG. 10M shows one condition which can be used to determine if the sieveopening should be closed, and this condition is a high amount of MOG asseen at the hopper.

FIG. 10N shows one condition which can be used to determine if the sieveopening should be opened up, and this condition is a high amount ofreturns.

XI. Membership Functions

In one embodiment, the main combine automation control system is a fuzzyinference system based on the cause/effect diagrams shown in FIGS. 10Athrough 10N. The system inputs 900 are mapped into fuzzy membershipfunctions as shown in Table 1 below. Then the outputs are mapped tofuzzy membership functions as shown in Table 2. Finally, several combineautomation rules are created to determine the behavior of the combineautomation system, as shown in Table 3.

TABLE 1 Mapping of System inputs Variable Input MF1 MF2 MF3 1 LookAheadlow ideal high 2 FeederTorque low ideal high 3 RotorTorque low idealhigh 4 EngineLoad low ideal high 5 ProcessorMADS ideal back 6ProcessorLoss low ideal high 7 ChafferLoss low ideal high 8 BlowingLosslow ideal high 9 ChafferMADS ideal back 10 ChafferMOG ideal high 11HopperMOG low ideal high 12 CrackedGrain low ideal high 13 Tailings lowideal high 14 Moisture dry ideal wet 15 Optimization loss groundSpeedcleanliness

TABLE 2 Mapping of System Outputs (the Combine Adjustments) VariableOutput MF1 MF2 MF3 1 Ground Speed low ideal high 2 Rotor Speed tooFastideal tooSlow 3 Concave tooClosed ideal tooOpened 4 Fan Speed tooFastideal tooSlow 5 Chaffer Opening tooClosed ideal tooOpened 6 SieveOpening tooClosed ideal tooOpened

TABLE 3 Combine Automation System Rules  1. If (LookAhead is high) or(FeederTorque is high) or (RotorTorque is high) or (ProcessorMADS isback) or (ProcessorLoss is high) then (GroundSpeed is high) (0.5)  2. If(LookAhead is low) or (FeederTorque is low) or (RotorTorque is low) or(ProcessorLoss is low) then (GroundSpeed is slow) (0.5)  3. If(EngineLoad is high) then (GroundSpeed is high) (1)  4. If (EngineLoadis low) then (GroundSpeed is slow) (1)  5. If (CrackedGrain is high)then (RotorSpeed is tooFast) (1)  6. If (ProcessorMADS is back) or(ProcessorLoss is high) then (RotorSpeed is tooSlow) (1)  7. If(Moisture is dry) then (RotorSpeed is tooFast) (0.5)  8. If (Moisture iswet) then (RotorSpeed is tooSlow) (0.5)  9. If (RotorTorque is high) or(EngineLoad is high) or (ProcessorLoss is high) or (ChafferLoss is high)or (ChafferMOG is ideal) or (CrackedGrain is high) then (Concave istooClosed) (1) 10. If (RotorTorque is low) or (EngineLoad is low) then(Concave is tooOpened) (1) 11. If (BlowingLoss is low) or (ChafferMADSis back) or (ChafferMOG is ideal) or (Tailings is high) then (FanSpeedis tooSlow) (1) 12. If (BlowingLoss is high) then (FanSpeed is tooFast)(1) 13. If (ChafferLoss is high) then (ChafferSpacing is tooClosed) (1)14. If (ChafferLoss is low) or (ChafferMOG is ideal) or (HopperMOG isideal) or (Tailings is high) then (ChafferSpacing is tooOpened) (1) 15.If (HopperMOG is high) then (SieveSpacing is tooOpened) (1) 16. If(Tailings is high) then (SieveSpacing is tooClosed) (1)XII. Alternative Embodiment Control Systems and GUIs (FIGS. 11-14)

FIGS. 11-14 show systems 1002 and 1052 comprising alternativeembodiments and modified aspects of the present invention with graphicaluser interfaces (GUIs) 1004, 1054 respectively for displaying MassMaterial Distribution (MMD) data and parameters graphically andreceiving operator inputs to facilitate an operator monitoring andcontrolling the combine or harvesting machine 500 in real-time. The GUIs1004, 1054 display “cause-and-effect” system operation monitoring inreal-time, based on operator input and system operating parametersrespectively. The systems 1002 and 1052 thus enable equipment operatorsto interactively monitor and control equipment, such as the combine 500.Moreover, the content of the GUIs 1004, 1054 can be displayed with anon-board monitor, a “smart phone,” a notebook or laptop computer, orother interactive mobile device, which can be either hard-wired orwirelessly connected, including via the “cloud” (Internet).

As shown in FIG. 11, the GUI 1004 can be configured for displaying bothnumerical and graphical representations of system operating parameters,which can be superimposed on a background image 1006 of the equipment,e.g., the combine 500. The GUI 1004 receives data (e.g., performanceparameters) generally originating from sensors mounted on the harvester500. An operator can customize the display output by interfacing withthe GUI 1004. E.g., configuring the GUI 1004 as a touchscreen enablesactivating various sensor output displays by touching specifictouchscreen areas. The touchscreen areas preferably depict harvester 500features and subsystems of interest whereby an operator has access viatouch points to system 1002 performance parameters, preferably inreal-time. For example, touch points on the engine area of the harvester500 image 1006 can activate displays of engine RPM, fuel consumption,manifold pressure, etc. Such displays can be representedalpha-numerically, graphically (e.g., a tachometer image), etc.Moreover, such performance and operational parameters 1016 can bedisplayed on the GUI 1004 as MMD graphs 1018 with values displayed overpredetermined time periods, thus indicating changes and cumulativeperformance as a function of the graph areas (e.g., areas undercorresponding curves).

The MMD data 1018 can be shown in real-time for operator use inmonitoring and controlling the equipment for optimum performance.Cumulative metrics can also be displayed, such as yield, moisturecontent, productivity (e.g., volume of harvested crop per acre), MOG,tailings, etc. Examples of performance metric displays are shown acrossthe top of the GUI 1004 and can include, without limitation, percentageof cracked grain 1008, percentage of MOG 1010, moisture contentpercentage 1012 and yield amount 1014. The amplitude dimension of theMMD data can correspond to such variables as harvested crop volume, fuelconsumed, acres covered, etc. The shape or profile of the MMD curve canindicate the efficiency and effectiveness of the combine 500. Theoperator inputs tend to alter such graph shapes and profiles, andcorrespondingly change graph amplitudes.

Harvester operating parameters affecting yields are shown at 1016 andcan include rotor speed, concave spacing, chaffer spacing and sievespacing. These parameters can generally be adjusted by the operator inthe field, and tend to directly affect the combine output andperformance, as shown by MMD graphs 1018, which can be transmitted toand displayed on an interactive mobile device, such as a smart phone, alaptop computer, etc., as discussed above. Such an interactive mobiledevice can be used to transmit operating instructions to the equipmentfor manipulating the operating parameters. The amplitude (volume) valuesof the MMD graphs can indicate amounts of material being processed bythe combine (harvester), while the MMD shapes or profiles can indicatethe efficiencies and effectiveness of the combines. Manually orautomatically adjusting the combine settings (i.e., operatingparameters) can result in changes to the system efficiencies andeffectiveness, which will be observable in real-time via changes in theMMD 1032 profiles and shapes.

A generally triangular pop-up window 1020 can be provided in the GUI1004 and can utilize touchscreen input technology for controllingoperating parameters 1016 in real-time. In the example shown, thetriangular display 1020 includes three harvesting parameters, i.e.,grain quality, combine speed, and grain loss, which are interrelated. Inother words, software algorithms generate displays superimposed on atriangular shape for identifying the parameters receiving the mostattention from the control system. For example, green can designate theareas that are receiving the most attention. A green control circle 1032a generally centered in the triangular user interface (UI) 1020, wouldindicate equilibrium with the three operating parameters generallyequally balanced. As shown in FIGS. 11 and 11A, the centered greencontrol circle 1032 a is surrounded by a yellow display area 1032 b,identifying areas of the harvesting machine that have had their controlthrottled or slowed to balance the green control circle 1032 a. At theperimeter of the triangular display 1020, red areas 1032 c indicateparameters 1016 that are receiving little or no attention from thecontrol system 1002. In the example shown in FIGS. 11 and 11A,relatively equal balance is being maintained by the control system amongthese parameters. The MMD displays 1032 a,b,c can indicate actualperformance parameters in real time vs. projected performanceparameters. The performance parameters 1016 are controlled by manualoperator adjustments, or automatically by the system 1002.

The MMD graphs 1018 shown to the right of the pop-up window 1020 reflectthe operating parameter adjustments based on such operator interface andinput. The MMDs 1018 can include horizontal lines indicating averagesettings. Moreover, first color (e.g., red) MMD displays can indicateactual performance, with a second color (e.g., green) showing idealperformance.

Using a touchscreen (or touchpad) slider 1022, the operator can controlwhich areas (corresponding to equipment operating parameters) arereceiving the most focused automation in a particular system operatingmode. For example, with the green control circle 1032 a in the lower,left corner (FIG. 12), quality will be maximized and loss minimized, butwith a slower combine speed. In this example the MMD graphs 1018 haveactual performance lines which are substantially horizontal due to thecontrol software being focused on one aspect, e.g., quality.

Conversely, manually adjusting the combine operating parameters canincrease speed, but generally compromises quality and increases grainloss. Data from the various sensors located on the machine provideinputs (e.g., the system inputs 900 shown in FIG. 8) to the combineautomation system 940, which can be displayed to the operator inreal-time via the MMDs 1018, which constantly change magnitude and shapebased on the actual operating parameters.

FIG. 13 shows an alternative GUI 1024 display (shown enlarged in FIG.13A). FIG. 13B shows the GUI of FIG. 13 with a pop-up window 1036showing an image of grain material in the combine (harvester). Camerasand other sensors can be placed at locations of interest in theharvester 500 and similar pop-up windows can be operator-activated byGUI touch points.

FIG. 14 shows another alternative embodiment comprising a systemdesignated 1052 with a modified GUI designated 1054. Common features ofthe alternative embodiment systems 1002, 1052 have common referencenumerals. The GUI 1054 includes a first (left) screen section 1056showing the MMD graphs 1018 and performance metrics, such as percentagesof cracked grain 1008, MOG 1010, moisture content 1012, yield/loss 1014,along with combine operating parameters 1016, such as rotor speed,concave spacing, chaffer spacing and sieve spacing.

A second (right) screen section 1058 shows the new or latest parametercontrol settings. The triangular control system 1020 and the controlcircle 1032 a of the previous embodiments can be replaced with selectionbars 1060 a for quality, 1060 b for MOG percentage and 1060 c for losspercentage. Control is accomplished through the selection bars 1060a,b,c, which include respective decrease and increase switch inputs 1062a,b respectively. The selection bars 1060 a,b,c can include respectivecenterlines 1064 a,b,c, indicating balanced or optimum target operatingparameter 1016 conditions.

The alternative embodiment GUI 1054 includes an automation enablementswitch 1066 with a display 1068 indicating Auto: ENABLED or Auto:DISABLED. When this function is enabled, the machine will beself-adjusting as long as material is flowing therethrough, with thecontrol system being configured with optimum operating parameters andautomatically maintain the operating parameter values. When disabled,the machine will return to the manual input mode of operation, requiringparameter 1016 input by the operator. Other than the operator inputs viathe selection bars 1060 a,b,c in lieu of the pop-up window 1020, theoperation of the GUI 1054 is similar to that of the GUI 1004 describedabove.

The output computed and provided by the processor to the control system940 can include system (i.e., combine) operating parameters. The GUI isconfigured for receiving the processor output. The GUI is alsoconfigured for displaying multiple colors each corresponding to arespective operating parameter.

Having described the preferred embodiments, it will become apparent thatvarious modifications can be made without departing from the scope ofthe invention as defined in the accompanying claims.

The examples and processes defined herein are meant to be illustrativeand describe only particular embodiments of the invention.

Having thus described the invention, what is claimed as a new anddesired to be secured by Letters Patent is:
 1. An automated combineincluding: a combine drive system; a crop material handling systemincluding a grain material intake at a front of the combine, an onboardreservoir configured for receiving grain material from the intake andretaining a quantity of grain material received from the intake and agrain material discharge configured for discharging grain material fromthe reservoir; said combine including: multiple operating parametersensors each sensing a respective combine operating parameter includingspeed, grain intake quantity, grain loss quantity and grain quality;said operating parameter sensors including a predictive sensor capableof sensing a respective operating parameter associated with said grainprior to said grain entering said combine; a control system configuredfor receiving input from said operating parameter sensors and providingoutput representing said operating parameters; a graphical userinterface (GUI) connected to said control system and configured fordisplaying said operating parameters; said GUI including a touchscreenfor operator input to said control system; said control systemconfigured for adjusting said combine operating parameters in responseto operator input via said touchscreen and displaying combine operatingparameter results of said combine operating parameter adjustments; andsaid operating parameter results associated with said display includecrop quality, combine speed and crop loss.
 2. The combine according toclaim 1, which includes: said GUI being configured for displayingmultiple colors each corresponding to a respective operating parameter.3. The combine according to claim 1 wherein: said GUI touchscreenincludes a pop-up window with a polygonal configuration, each polygonside corresponding to a respective operating parameter; and said pop-upwindow includes a slider input button configured for operatormanipulation emphasizing and deemphasizing respective operatingparameters associated with said pop-up window.
 4. The combine accordingto claim 3, which includes: said pop-up window including a centralizedcontrol circle with a first color depicting a first level of controlsystem emphasis, said control circle including said slider input button,a middle area surrounding said control circle and depicting a secondlevel of control system emphasis and a perimeter surrounding said middlearea and depicting a third level of control system emphasis; and saidcontrol system displays colors representing corresponding levels ofcontrol system attention devoted to each operating parameter.
 5. Thecombine according to claim 4, which includes: an operating parameterrepresenting a Mass Material Distribution (MMD) corresponding to one ormore of material volume, combine efficiency and combine effectiveness.6. The combine according to claim 1, which includes: said GUIgraphically displaying a current balance with a polygonal display havingmultiple display sides each representing a respective combine operatingparameter.
 7. The combine claim 6, which includes a central location insaid polygonal display wherein a respective color display indicatesbalance of operating parameters optimizing combine performance.
 8. Thecombine according to claim 1, which includes: said control system beingconfigured with optimum operating parameter values and automaticallymaintaining said operating parameter values with said combine inoperation.
 9. The combine according to claim 1, which includes: said GUIincluding a first curve representing a projected operating parameter anda second curve representing an actual operating parameter displayed inreal-time.
 10. The combine according to claim 1 wherein said GUIincludes: a first output display screen section showing MMD graphs andcombine performance including one or more of operating parameters chosenfrom the list consisting of: cracked grain percentage, MOG percentage,moisture content, loss percentage, rotor speed, concave spacing, chafferspacing, and sieve spacing; and a second output display screen sectionshowing the latest combine operating parameter control settings.
 11. Thecombine according to claim 10, which includes: said second outputdisplay screen section including displays indicating broken grainpercentages, MOG percentages and grain loss percentages; and said secondoutput display screen section including increase and decreasetouchscreen inputs for controlling said broken grain percentages, MOGpercentages and grain loss percentages.
 12. The combine according toclaim 1, which includes: look-ahead and look-aside crop mass predictivesensors connected to said control system; wherein the look-ahead andlook-aside crop mass predictive sensors are capable of providing anestimate of the mass of the crop that is immediately in front of andalongside the combine respectively before it actually enters thecombine; and wherein the estimate of the mass of the crop is provided tothe control system as an additional input for optimizing combineperformance.